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ABSTRACT 



This study reviews selected aspects of the evaluation 
process for the American Council on Education-Kellogg Project on Leadership 
and Institutional Transformation. Attention is focused on developing an 
understanding of institutional transformation and the "lessons learned" about 
the various methods and strategies used. Institutions were selected for the 
project based on: breadth of campus involvement in the planning process; 
clarity of the institutions ' s vision and goals; extent to which the 
institution was ready to and capable of implementing its change agenda; and 
degree of recognition of the role and impact of institutional structures in 
accomplishing change. Project information sources included: institutional 
self -ref lective reports and updates; "homework" activities completed by 
project teams in preparation for project meetings; reports from consultants 
who worked with each institution; correspondence with the agency and with 
consultants; and summary notes from project meetings. The information sources 
contained structured, open-ended questions which collected similar 
information about each institution, thus allowing inter- institutional 
comparisons. Data collection and analysis was a process carried out over four 
points in time, with teams selectively and progressively articulating and 
evaluating each set of lessons learned. Data analysis efforts resulted in a 
set of propositions that reflect strategies central to effecting 
institutional change. (SW) 
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This document provides an overview of select aspects of the evaluation process for the 
ACE/Kellogg Project on Leadership and Institutional Transformation. The project had two explicit 
components. The first and primary objective was to help institutions make progress on their self- 
defined, institutional transformation agendas. The second was to capture and understand the 
change process from the collective experiences of 26 diverse institutions so that other institutions 
might benefit. The original evaluation was concerned with both elements, but this paper focuses 
specifically on the challenges and methods related to developing an understanding of the methods 
and strategies used by project participants to effect institutional transformation. 

To understand the transformation process at 26 institutions we needed to design a data 
collection procedure which allowed us to make inferences across cases, focus on the processes, 
strategies and methods of institutional change, and benefit from the structure of the project and the 
informed perspectives of a range of team members, ACE staff, and senior liaisons (informed 
consultants who individually worked with each institution). 



The Challenges 

Any project attempting to better understand a complex phenomenon faces a host of 
challenges it must resolve. For this project we had to design a process which best minimized the 
following problems faced in collecting and analyzing the information: 

A large diverse group of institutions 

As noted above, the primary purpose of this project was to help a group of diverse 
institutions make progress on their institutional transformation agendas. Thus institutions were 
selected which were thought to be in positions able to effect change rather than upon criteria for 
maximizing research results. 

To arrive at our sample of 26 institutions, an invitation for proposals was sent to 450 ACE 
member institutions. From that invitation, 1 10 applications were submitted. A selection committee 
of ACE staff, staff from other higher education associations, past college presidents, and directors 
of similar national projects selected 26 institutions from that pool. Institutions were selected by 
the following 1 1 criteria based upon their submitted applications: 

• breadth of campus involvement in the planning process; 

• clarity of the institution’s vision and goals; 

• breadth of conceptualization and integration of the components of the change agenda; 

• extent to which the proposed change themes connected to the institution’s mission and 
identity; 

• clarity of perception of what is driving the change and how these forces relate to the 
institutional mission and strengths; 

• extent to which the institution has assessed its assets and liabilities, threats and 
opportunities in pursuing the change agenda; 

• evidence of boldness, imagination, and excitement about change; 

• extent to which the institution is poised and has the capacity to implement its change 
agenda; 

• degree of recognition of and attention to the developmental needs of those involved in 
the change process; 

• degree of recognition of the role and impact (positive and/or negative) of institutional 
structures and processes in accomplishing change; 
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• diversity of institutions. 

Twenty-six institutions (5 research universities; 7 doctoral universities; 5 comprehensive 
institutions; 5 liberal arts colleges; and 4 community colleges) of which 8 are private and one is a 
public HBCU, were selected which represented the diversity of American higher education and 
were thought ready to make headway on institutional transformation agendas. 

Different change agendas 

One element which differentiates the ACE Project from others was that it involved 
institutions working on diverse change agendas rather than a common agenda (such as enhancing 
technology or implementing Responsibility Centered Management). This hallmark of the project 
complicated the process for collecting information on change, as institutions were engaging in 
strategies to effect different types of institutional change. The common element around which the 
project was designed was not the content of the change, but rather the magnitude of the change — 
transformational change which has depth, breadth, and pervasiveness. 

Starting at different places in their agendas 

In addition to diverse institutions working on a variety of change initiatives, institutions 
entered the project at different stages of the formulation and implementation of their campus 
agendas. At one end of the spectrum was a group of institutions that had been working on their 
transformational change agendas for many years. At the other end were institutions which began 
their efforts when they started the project. Still other institutions fell between the two poles. This 
created a challenge collecting information from institutions which were at different places in their 
transformation agendas. 

The projects were in motion 

Additionally, this project faced the challenge of collecting information while the campus- 
based projects were “in motion.” Institutions were in the thick of implementing their 
transformation agendas, yet they were attempting to assess their strategies and methods which had 
not yet produced many results. This created problems in trying to anchor processes to outcomes, 
which in turn made it difficult to determine the value of the approaches. 

Not directly observable nor consistently accessible information 

Finally, the project did not have researchers on each campus to capture or record what was 
occurring. The project rather had to rely upon information provided by the team members and 
collected at periodic visits to campus. By not directly observing the processes the information 
collected became open to bias and inconsistency within and across institutions. 

The described elements set limitations which prevented us from developing a data collection 
and analysis process which 1) focused on specific changes (such as curricular change), 2) included 
a process to assess the progress of each institution, 3) identify common strategies at a high level of 
specificity, or 4) determine the value added by the project 



Methods for Collecting and Analyzing Lessons Learned 

To designed a data collection and analyses process that addressed the above challenges we 
developed the following process. 

A focus on lessons learned 

We felt the best way to capture the experiences of the 26 participating institutions in ways 
which others might benefit was to focus on the lessons institutions were learning through their 
efforts with various strategies and methods. The lessons learned would be based upon the 
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cumulative learning from project teams that was generated by their experiences and trials as well as 
from the lessons the senior liaisons were learning through their experiences working with clusters 
of institutions. Because of the complexity of the project, the diversity of institutions working on 
different issues and starting at different places, and our distance from the change efforts, lessons 
learned became our data collection centerpiece. Lessons learned (1) were easily captured in 
reflective reports, (2) allowed us to benefit from the collective experiences of informed insiders 
who were working with clusters of institutions, both project team members and senior liaisons; (3) 
incorporated the language of the participants into the findings; (4) were able to capitalize on both 
the successes and failures of project team strategies; and (5) were in a format that other institutions 
might easily use. 

From the desire to understand the lessons institutions were learning about transformation 
we developed the following question to guide our process: 

What general lessons about effecting intentional transformation can be identified from the 

experiences of 26 institutions? 

Our data collection and analysis methods were designed to address this question. Our goal 
at the completion of the project was to have identified a list of Project Lessons that institutions 
engaged in transformation might find helpful that were drawn from the experiences of project 
participants. The following description provides the road map explaining how we developed 
Project Lessons. 

Data collection points: 

The evaluation process was designed to encompass four measurement points in time so that 
the information was systematically and continuously collected. 

The Baseline. The Baseline measurement was the starting point for data collection. The 
purpose of the Baseline was to help institutions begin the reflective process required over the 
course of the project, articulate their comprehensive change agenda, and think intentionally about 
the process of change. It also asked them to outline their intended strategies and methods to affect 
comprehensive change. The Baseline did not collect information pertinent to lessons about how to 
effect change, thus it will not be discussed further in this document. The role of the baseline was 
more salient to the primary objective of the project, helping 26 institutions make progress on their 
change agendas. 

Mid-Point 1 and 2. The next two collection points occurred during the middle of the 
project. The project had two Mid-Points at which information on the institutional change process 
was collected. Each collection point coincided with a project meeting — Mid-Point 1 occurred in 
conjunction with the October 1996 Project Meeting and Mid-Point 2 coincided with the March 
1997 Project Meeting. The Mid-Points helped institutions focus their attention on the processes, 
methods and strategies they were using to effect comprehensive change and to begin articulating 
what they were learning from their efforts. It was originally hoped that both Mid-Points would 
provide adequate information to collect and analyze lessons about change. Only after the 
information at Mid-Point 1 was gathered and analyzed did we realize that we were premature in 
collecting information about lessons learned. Project teams simply had not had adequate time to 
implement strategies, evaluate their results, and articulate lessons. Thus only Mid-Point 2 is 
discussed further. 

The End-Point. The final information collection point was the End-Point. This was the 
terminal point for the project, although for most participating institutions it was not the completion 
point for their change agendas. The End-Point allowed ACE to focus extensively on what 
institutions have done to effect comprehensive change and the lessons they have learned through 
their experiences. 
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Sources of Information: 

Information over the course of the project was gleaned from the following sources: 

• Institutional self-reflective reports/updates; 

• “Homework” activities completed by project teams in preparation for project meetings; 

• Senior liaison reports from campus visits; 

• Informal institutional self-reports, updates and correspondence with ACE staff and 
senior liaisons; 

• Summary notes from discussions at project meetings. 

The table below shows the sources of information by the point in the project at which it was 
collected. Each source of information obtained from institutions and senior liaisons became a unit 



of analysis. 








BASELINE 


MID-POINT 1 


MID-POINT 2 


END-POINT 


First Inst. Report 


Inst. Update 


Inst Update 


Final Inst Report 


Sr. Liaison Update 


Sr. Liaison Update 


Sr. Liaison Update 


Sr. Liaison Update 






Project Meeting 
“Homeworic” 


Fall Forum 

“Sense of Meeting Notes’ 






Meeting Summaries 


Meeting Summaries 




Mid-Point 2 


Data Collection and 


Analysis 



Data Collection 

Mid-Point 2 was the initial period at which the project began focusing on the lessons that 
institutions were learning about effecting transformation. Prior to this point, most institutions had 
not been working on their transformation agendas long enough to have reflected upon their 
strategies and methods and learned lessons about what they were accomplishing (and not 
accomplishing). Information at Mid-Point 2 was obtained from the following sources: 

1 . The project teams reflected upon their progress and learning through a requested Mid- 
Point 2 update; 

2. Project teams completed reflective “homework” in preparation for the Spring Project 
Meeting; 

3. The senior liaisons provided reports from campus visits; 

4. The ACE staff, senior liaisons, and selected team members wrote summaries of the 
discussions at the Spring Project Meeting. 

The mid-point updates, the project meeting homework and the senior liaison reports 
contained structured, open-ended questions which collected similar information about each of the 
institutions allowing for comparisons across cases and within institutions across data sources 
(Miles & Huberman, 1994). These three documents, along with summaries of discussions from 
the Spring Project Meeting, converged on similar questions to provided triangulated data to ACE, 
enabling us to collect data at multiple points and cross-check our assumptions and insights (Patton, 
1990). 



By Mid-Point 2, we were also able to distinguish between the progress of institutions 
which allowed us to assign various weights to their lessons. To determine which institutions were 
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making progress and which were not, the senior liaisons who were intimately familiar with the 
institutions and ACE Project staff classified the institutions into one of three categories: 1) moving 
forward, 2) moving somewhat, and 3) stuck. This classification allowed us to add weight to 
various lessons collected from institutional participants. For example, the insights about making 
progress from institutions that we knew were moving forward were given increased weight, while 
the lessons about progress from institutions deemed not making progress were assigned less 
weight, and in some cases disregarded. 

Data Analysis 

Data analysis is at the heart of qualitative research; it is the process through which the data 
collected begins to take shape, form patterns and trends, and make sense. To begin analyzing the 
first set of collected data a three step process was followed (Miles and Huberman, 1994). First, 
the initial data which focused on lessons institutions were learning (institutional mid-point updates, 
senior liaison fall visit reports, project meeting homework, and summaries from the spring project 
meeting) was reviewed and pattern coded, a process which linked bits of data into “a smaller 
number of sets, themes or constructs” (p. 69). Second, we engaged in memoing, a process of 
writing up ideas and notions of the pattern coded data which “tie pieces of data together into a 
recognizable cluster to show that those data are instances of a general concept” (p. 73). Third, we 
developed a series of propositions from the memos attempting to capture the strategies central to 
effecting comprehensive institutional change. 

After an initial draft of propositions was created, we invited the senior liaisons and an 
outside reviewer to read and comment on them. We asked them if the propositions made intuitive 
sense, if the experiences of the senior liaisons were consistent with the spirit of the propositions, 
and where the propositions might be modified and enhanced. Based upon the obtained feedback a 
second generation of propositions was drafted. 

A second set of data was then collected and analyzed. The second data consisted of 
updates written by the senior liaisons describing their spring site visits as well as institutional self- 
reports and informal memos to ACE from various project teams highlighting their experiences. 
This data was additionally pattern coded. Two researchers independent of one another then 
engaged in memoing to link together the pattern coded data. By doing so independently, 
researcher bias was reduced (Patton, 1990). The two sets of newly created memos were then 
incorporated into the most recent set of propositions, creating two new sets of propositions, one 
from each researcher. The two drafts were then compared and reconciled creating a third draft of 
propositions. 



End-Point Data Collection and Analysis 



Data Collection 

To continue the data collection and analysis process and refine the propositions, we sought 
additional data at the End-Point which was then incorporated into the propositions modifying, 
expanding, confirming and disconfirming our previous notions. The End-Point is the final phase 
of our evaluation process and the one which carried the most weight (Although this paper is 
written in past tense, this final phase of data collection has yet to be completed at this writing. 
Thus, the design below is our intended process and may differ slightly from the actual final 
process.) 

A three component End-Point process was designed to occur September 1997 through 
January 1998. The purpose of this final phase was to help institutions assess their progress, plan 
for their next steps at the conclusion of the ACE Project, and most importantly reflect upon what 
they have learned. At the End-Point: 
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1 . Each team was asked to complete the End-Point Report which was to be written as a 
stand-alone document The final report focused on (a) results and progress on their 
institutional transformation agendas (including initial expectations and actual 
outcomes), (b) reflections on the change process (including their lessons learned), and 
(c) thoughts and ideas on the concept of institutional transformation. 

2. Each team was asked to use the final senior liaison visit to conduct a Campus Forum on 
Transformation. Composed of 25-30 people, each forum was designed to be a 
campus-wide discussion on the change process to help project teams 1) focus on their 
impact, 2) reflect on their processes and extract lessons they have learned through their 
experiences, and 3) plan for “life-after-ACE.” From the campus forums, a “sense of 
meeting notes” was crafted capturing the discussion with specific attention to lessons 
learned. 

3. The Final Project Meeting (January 1998) was designed to focus on lessons learned 
and confirmed. This meeting consisted of a series of discussions on lessons learned 
and the propositions drafted to date. By involving project team members to comment 
on the propositions, those most intimately involved with comprehensive change were 
allowed to give feedback, criticism, and clarity to the propositions which informed a 
next draft. 

Data Analysis 

The final data analysis procedure included reviewing four documents. From the three 
component End-Point process three documents were produced and a summary of the discussions 
at final project meeting was created. First, “sense of meeting” notes capturing the conversation at 
the Campus Forums on Change was collected. Second, a final report from each senior liaison 
highlighting their final visit and giving their views on each campus forum was obtained. Third a 
final report was drafted by each project team. Following a similar process, these documents were 
read and the data pattern coded and memoed (Miles & Huberman, 1994). The new memos were 
then compared to the latest draft of propositions, modifying expanding, clarifying, confirming and 
discontinuing so that a final set of propositions might be created. (Because this process is yet to 
be completed, we are unsure if a single researcher will finish this process or if the luxury of 
multiple researchers working independently to reduce researcher bias (Patton, 1990) will be 
followed.) 

The propositions were presented to project team participants at the final project meeting. 
From discussions of these propositions, meeting summaries were drafted. The information 
obtained from the meeting summaries were then analyzed. At this point the propositions became 
the first draft of Project Lessons. The Project Lessons were then distributed to senior liaisons and 
other experts on institutional transformation and asked for comments. From this final feedback we 
created a final set of Project Lessons. 



Methodological Limitations 

No research project is without its methodological limitations, it is what makes a project 
manageable (Leonard-Barton, 1995). At the same time, researchers can engage in a number of 
strategies to ensure acceptable levels of methodological rigor. Many of limitations originated from 
the difficulties identified early in the project and outlined at the beginning of this paper. Even 
though solutions were presented in the above descriptions of the data collection and analyses 
processes not all of the problems were addressed. The following limitations are those which could 
not be readily solved and may influence the Project Lessons. 
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The campus change initiatives are incomplete 

First, most of the participating institutions’ change agendas are incomplete. Because of the 
size and the scope of each institution’s transformation efforts, project teams may not have 
observable evidence for some time. Limited results create difficulties linking processes to 
outcomes. At the same time, strategies which initially showed positive results may over a longer 
period lead to harmful effects. The same is true for strategies which initially failed as they may 
from the perspective of a longer time frame be seen as beneficial. 

Information is self-reported 

The bulk of the materials we collected were provided by the institutions themselves which 
can be limiting because of a number of factors. First, respondents, when reporting data on 
organizational processes tend to hold more prominently in their minds events which occurred more 
recently than in the distant past (Kanter, 1983). They judge the value of events, not based upon 
their impact or a cause and effect pattern, but by chronological proximity (Glick, Huber, Miller, 
Doty, & Sutcliff, 1995). Second, people tend to jump to conclusions about cause and effect; rarely 
do they attribute good results to poor analyses or incorrect actions, view good actions as having no 
impact, or find poor results coming from good actions or accurate perceptions (Starbuck & 
Milliken, 1988). Third, informants may provide information which is incorrect or revisionist in 
nature (Leonard-Barton, 1995; Van Maanen, 1979). They may intentionally give information 
which is inaccurate or misleading because they want to hide controversial practices or decisions, or 
not reveal personal defects, mistakes or taboo activities (Van Maanen, 1979). This explanation 
may be specifically relevant in this project because institutions are participating in a highly visible 
national project. Informants may also unwittingly provide false information because they either 
were misled or given poor information by others, or because they were unaware of certain aspects 
of their own activities. (Van Maanen, 1979). 

Cause and effect relationships are difficult to determine 

The process of institutional change is a complex and not well understood phenomenon. In 
this project we asked institutions and those working with institutions (e.g., senior liaisons) to 
reflect and comment on a complex process. We asked change agents to draw conclusions and 
lessons about strategies to effect change when the links between cause and effect were tenuous at 
best and indeterminable at worst. The difficult in linking outcomes to processes may lead to 
“superstitious learning’’ (March & Olsen, 1975) as people attribute outcomes to the wrong 
processes. 

Loose kev insights throug h generalization 

In this study, we attempted to draw generalizable lessons from a group of 26 diverse 
institutions working to implement diverse change initiatives. Our intent was to develop a set of 
Project Lessons informative to others desiring to implement transformative change. In our efforts 
to draw conclusions we most likely lost some key specific strategies that individual institutions 
used to effect change. We could not document all of the individual approaches or their nuances 
from each institution, and thus were unable to capture the detail that might have been important, but 
only so at a single institution. 

Biased hv the structure of the project 

Finally, the findings might be biased because of the project design. The primary intent of 
this project was to help institutions progress on their institutional transformation agendas. By 
meeting the first project goal — helping institutions make progress on their change agendas — we 
most likely created an unnatural environment that prevented us from rigorously meeting our second 
goal — learning lessons that other institutions can use. (For example, lesson number one might 
be; join a national project which includes help from consultants.) We worked with institutions at 
numerous levels, such as at project meetings and through site visits with senior liaisons, gave them 
materials and information on institutional change, all interventions which (we hoped) facilitated 
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their processes. The involvement by ACE may have shaped their strategies and thus affected their 
lessons. 

Nevertheless, we feel confident that our Project Lessons were developed in a manner 
which captures the collective experiences of those involved. We stand by their rigor and hope that 
others find value. 
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